Adversarial Training Data Augmentation for Generating Related Responses

ABSTRACT

An intelligent computer platform to introduce adversarial training to natural language processing (NLP). An initial training set is modified with synthetic training data to create an adversarial training set. The modification includes use of natural language understanding (NLU) to parse the initial training set into components and identify component categories. As input is presented, a classifier evaluates the input and leverages the adversarial training set to identify the intent of the input. An identified classification model generates accurate and reflective response data based on the received input.

BACKGROUND

The present embodiments relate to an artificial intelligence platformand a feature extraction technique. More specifically, the embodimentsrelate to training augmentation data for text classification and intentgeneration.

In the field of artificially intelligent computer systems, naturallanguage systems (such as the IBM Watson™ artificially intelligentcomputer system or and other natural language question answeringsystems) process natural language based on knowledge acquired by thesystem. To process natural language, the system may be trained with dataderived from a database or corpus of knowledge relating to thepeculiarities of language constructs and human reasoning.

Machine learning (ML), which is a subset of Artificial intelligence(AI), utilizes algorithms to learn from data and create foresights basedon this data. AI refers to the intelligence when machines, based oninformation, are able to make decisions, which maximizes the chance ofsuccess in a given topic. More specifically, AI is able to learn from adata set to solve problems and provide relevant recommendations.Cognitive computing is a mixture of computer science and cognitivescience. Cognitive computing utilizes self-teaching algorithms that usedata minimum, visual recognition, and natural language processing tosolve problems and optimize human processes.

At the core of AI and associated reasoning lies the concept of textclassification, which is an area of natural language processing (NLP)that focuses on labeling and organizing text. A natural languageclassifier service applies cognitive computing techniques to return bestmatching predefined classes for short text inputs, such as a sentence ora phrase. The text inputs are expressed in natural language and placedinto categories. The classifier returns a prediction of a class thatbest captures what is being expressed in the associated text. Based onthe predicted class, an application can be leveraged to take anappropriate action, such as provide an answer to a question, suggest arelevant product based on expressed interest, or forward the text inputto an appropriate venue. Accordingly, natural language understandingclassifiers evaluate natural language expressions, place the expressionsinto categories, and return a corresponding classification.

The process of understanding natural language requires reasoning from arelational perspective that can be challenging. Structures, includingstatic structures and dynamic structures, dictate a determined output oraction for a given determinate input. The determined output or action isbased on an express or inherent relationship within the structure. Thisarrangement may be satisfactory for select circumstances and conditions.However, it is understood that dynamic structures are inherently subjectto change, and the output or action may be subject to changeaccordingly. Existing solutions for efficiently identifying andunderstanding natural language and processing content responses to theidentification and understanding as well as changes to the structuresare extremely difficult at a practical level.

SUMMARY

The embodiments include a system, computer program product, and methodfor natural language content processing, including natural languageunderstanding and content distribution processing.

In one aspect, a computer system is provided with a processing unitoperatively coupled to memory, and an artificial intelligence (AI)platform to support natural language processing. A tool in the form of aclassifier is provided with the AI platform. The classifier uses naturallanguage understanding (NLU) to evaluate and process received inputagainst an adversarial training set. The classifier leverages theadversarial training set to predict a classification label, which isthen used to identify a classification model. The classification modelidentifies both intent and a corpus corresponding to the input. Usingthe input as characteristic data, the classification model producesresponse data that reflects the received input.

In another aspect, a computer program product is provided with acomputer readable storage medium having computer readable program codeembodied therewith, the program code being executable by a processor tosupport natural language processing (NLP). Program code is provided toevaluate received input against an adversarial training set usingnatural language understanding (NLU). Program code leverages theadversarial training set to predict a classification label correspondingto the input, which is then utilized by the program code to identify aclassification model. The classification model identifies acorresponding intent and corpus. Program code executes theclassification model using the input as characteristic data, andresponse data is produced to reflect the received input.

In yet another aspect, a method is provided with an artificialintelligence (AI) platform for processing natural language. Receivedinput is evaluated using natural language understating (NLU). Anadversarial training set is leveraged to process the evaluated input andidentify a classification model. Execution of the classification modelidentifies the corresponding intent of the input and a corpuscorresponding to the model. The classification model uses the input ascharacteristic data to produce response data that reflects the receivedinput.

These and other features and advantages will become apparent from thefollowing detailed description of the presently preferred embodiment(s),taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The drawings reference herein forms a part of the specification.Features shown in the drawings are meant as illustrative of only someembodiments, and not of all embodiments, unless otherwise explicitlyindicated.

FIG. 1 depicts a system diagram illustrating a schematic diagram of anatural language process system to provide context to word vector anddocument vector representations.

FIG. 2 depicts a block diagram illustrating the tools shown in FIG. 1and their associated application program interfaces (APIs).

FIG. 3 depicts a flow chart illustrating a process for generatingsynthetic utterances to expand training data.

FIG. 4 depicts a diagram illustrating an example lattice graph.

FIG. 5 depicts a flow chart illustrating an adversarial training processto leverage the synthetic utterances formed in FIG. 4.

FIG. 6 depicts a flow chart illustrating an exemplary process formanaging evolution of the classification model shown and described inFIG. 5.

FIG. 7 depicts a flow chart illustrating application of the intentclassification model.

FIG. 8 is a block diagram illustrating an example of a computersystem/server of a cloud based support system, to implement the systemand processes described above with respect to FIGS. 1-7.

FIG. 9 depicts a block diagram illustrating a cloud computerenvironment.

FIG. 10 depicts a block diagram illustrating a set of functionalabstraction model layers provided by the cloud computing environment.

DETAILED DESCRIPTION

It will be readily understood that the components of the presentembodiments, as generally described and illustrated in the Figuresherein, may be arranged and designed in a wide variety of differentconfigurations. Thus, the following details description of theembodiments of the apparatus, system, method, and computer programproduct of the present embodiments, as presented in the Figures, is notintended to limit the scope of the embodiments, as claimed, but ismerely representative of selected embodiments.

Reference throughout this specification to “a select embodiment,” “oneembodiment,” or “an embodiment” means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiments. Thus, appearances of thephrases “a select embodiment,” “in one embodiment,” or “in anembodiment” in various places throughout this specification are notnecessarily referring to the same embodiment.

The illustrated embodiments will be best understood by reference to thedrawings, wherein like parts are designated by like numerals throughout.The following description is intended only by way of example, and simplyillustrates certain selected embodiments of devices, systems, andprocesses that are consistent with the embodiments as claimed herein.

Changes to the input of a neural network affect output. Adversarialexamples are inputs to a neural network that result in an incorrectoutput from the network. Systems that incorporate deep learning modelsutilize training data for image and text classification. To make thedeep learning models more robust adversarial training is introduced. Asshown and described below, a system, method, and computer programproduct is provided to combine perturbation based data augmentation withadversarial training. The perturbation is directed at application of anon-complex textual transformation to one or more training sentences.Synthetic data is created from training sentences by replacement ofselect sentence components, such as terms and paraphrases. A selectsubset of the synthetic data is utilized in the adversarial training ofthe classification model as a member of the model training data.Accordingly, the adversarial training not only creates synthetic data,but selects a subset of the synthetic data having an adversarialcharacteristic, and assigns that subset of synthetic data as trainingdata for the classification model.

Referring to FIG. 1, a schematic diagram of a computer system (100) isdepicted with a generalized adversarial training framework for textclassification. The computer system (100) is configured to train acorresponding intent model classifier and utilize the trained classifierto predict a classification label of received input. As shown, a server(110) is provided in communication with a plurality of computing devices(180), (182), (184), (186), and (188) across a network connection (105).The server (110) is configured with a processing unit (112) incommunication with memory (116) across a bus (114). The server (110) isshown with an artificial intelligence (AI) platform (150) for documentand context processing over the network (105) from one or more computingdevices (180), (182), (184), (186) and (188) via communication paths(102) and (104). More specifically, the computing devices (180), (182),(184), (186), and (188) communicate with each other and with otherdevices or components via one or more wired and/or wireless datacommunication links, where each communication link may comprise one ormore of wires, routers, switches, transmitters, receivers, or the like.In this networked arrangement, the server (110) and the networkconnection (105) may enable content and/or context recognition andresolution for one or more content users. Other embodiments of theserver (110) may be used with components, systems, sub-systems, and/ordevices other than those that are depicted herein.

The AI platform (150) may be configured to receive input from varioussources. For example, AI platform (150) may receive input from thenetwork (105), one or more knowledge bases (160) of electronic documents(162) or files (164), or other data, content, and other possible sourcesof input. In selected embodiments, the knowledge base (160), alsoreferred to herein as a corpus, may include structured, semi-structured,and/or unstructured content in a plurality of documents or files. Thevarious computing devices (180), (182), (184), (186), and (188) incommunication with the network (105) may include access points forcontent creators and content users. Some of the computing devices(180)-(188) may include devices to process the corpus of data withrespect to word vector generation, thereby enhancing natural languagebased services. The network (105) may include local network connectionsand remote connections in various embodiments, such that the AI platform(150) may operate in environments of any size, including local andglobal, e.g. the Internet. Additionally, the AI platform (150) serves asa front-end system that can make available a variety of knowledgeextracted from or represented in documents, network accessible sourcesand/or structured data sources. In this manner, some processes populatethe AI platform (150), with the AI platform (150) also including inputinterfaces to receive requests and respond accordingly.

As shown, content may be in the form of one or more electronic documents(162) or files (164) for use as part of the corpus (160) of data withthe AI platform (150). The knowledge base (160) may include anystructured and unstructured documents or files (162) and (164),including but not limited to any file, text, article, or source of data(e.g. scholarly articles, dictionary, definitions, encyclopediareferences, and the like) for use by the AI platform (150). Contentusers may access the AI platform (150) via a network connection or aninternet connection to the network (105), and may submit naturallanguage input to the AI platform (150) that may effectively beprocessed for intent and corresponding response data. As furtherdescribed, the intent classification model functions to identify andextract features within an associated document or file corresponding toa received request and associated request data.

The AI platform (150) is shown herein with the tools to support andenable both application and augmentation of an intent classificationmodel. In one embodiment, an initial intent classification model isprovided. The tools of the AI platform (150) include, but are notlimited to, a trainer (152) and a classifier (154). The trainer (152)functions as a management tool of the intent classification model,hereinafter referred to as the model, including management of modelaugmentation. The classifier (154) functions as a tool to leverage themodel in order to process received data, including application of aclassification label, e.g. intent, to the received input, and to returnaccurate response data (170), which in one embodiment corresponds to theintent. The trainer (152) and the classifier (154) both address intentof a corresponding request, with the trainer (152) maintaining and/ormanaging the model, and in one embodiment dynamically updating the modelresponsive to the received request, and the classifier (154) to leveragethe model to provide accurate response data. The model (158) is shownherein stored local to the AI platform (150), although this locationshould not be considered limiting. In one embodiment, the model (158)may be stored on a remote storage device operatively coupled to theserver (110). Similarly, although only one classification model (158) isshown herein, it is understood that the system may include a pluralityor library of models (158), and as such a singular representation of themodel (158) should not be considered limiting. Response data (170) isgenerated by application of the model (158), and may be communicated toone or more of the systems (180)-(190) across the network connection(105). Similarly, in one embodiment, the generated response data (170)may be presented on a visual display (172) operatively coupled to theserver (110).

As briefly described above, the model (158) is leveraged to process areceived request with respect to classifying the intent of the request,which in one embodiment directs the classifier (154) to an appropriatelibrary within the knowledge base (160) to process the request. Thetrainer (152) functions as a tool to manage the model (158), and morespecifically manage augmentation of the model (158). As described indetail below, synthetic data is utilized to facilitate augmentation ofthe model (158). The trainer (152) is either provided with the syntheticdata or creates the synthetic data. In one embodiment, the trainer (152)is a natural language processing tool and utilizes natural languageunderstanding (NLU) to parse training data into grammatical components,and leverage a paraphraser or a paraphrase database, to create thesynthetic data as related to the training data. The trainer (152)identifies synthetic data as either closely or tangentially relatedgrammatical words or phrases of the training data. The adversarialaspect of the training is directed at selection of a subset of thesynthetic data, and the merge of the synthetic data subset into themodel (158). In one embodiment, it is the characteristic of thesynthetic data subset and selection of this subset that introduces theadversarial characteristics into the training data of the associatedmodel (158). Accordingly, the trainer (152) merges the synthetic datasubset with real training data to effectively introduce extrainformation, e.g. noise, thereby adding robustness to the model (158).

The model (158) is subject to training so that it may adapt to thedynamic nature and characteristics of data and data processing. Althoughonly one model (158) is shown herein, in one embodiment, the knowledgebase (160) may include a library (166) of models (166A), (166B), . . .(166N), with different models directed at different subjects and/orcategories. The trainer (152) receives initial training data, alsoreferred to herein as a training data set. In one embodiment, thetraining data set is received from one or more of the devices(180)-(190) across the network connection (102). The training data setmay come in different formats. The trainer (152) uses natural languageunderstanding (NLU) to parse the training data into sub-components andidentify a category for each parsed sub-components. The parsing supportsidentification of terms. More specifically, the trainer (152) uses theidentified categories to further identify paraphrase terms for thesub-components. Accordingly, the initial aspect of the functionality ofthe trainer (152) is to process the training set into categories andidentify corresponding paraphrase terms.

The trainer (152) creates or builds synthetic phrases, also referred toherein as utterances, with aspects of the training data and theparaphrase terms. More specifically, the trainer selectively replacesthe parsed sub-components with the paraphrase terms, with thereplacement creating synthetic data, as shown and described in FIGS.3-7. The synthetic data represents elements of the initial training dataset selectively augmented with the paraphrase terms. In one embodiment,the replacement paraphrase terms represent textual disturbances, e.g.perturbations, of initial training data. It is understood that a subsetof the synthetic data may be redundant or syntactically incorrect,collectively referred to as irrelevant synthetic data. In oneembodiment, low value synthetic data is removed from the synthetic dataset. Examples of the low value synthetic data include, but are notlimited to, a common word and/or a miss-spelled word. Accordingly, thetrainer (152) removes irrelevant and low value synthetic data from theformed set of synthetic data.

As shown and described, the synthetic data set is a set of data relatedto the training data and created through term replacement, paraphrasing,etc. The synthetic data set is not to be confused with adversarialtraining or adversarial data, as described in detail below. As shown inFIG. 4, a lattice graph structure comprised of the initial training dataset of the textual perturbations may be utilized to create one or moresynthetic phrases or sentences related to the training sentence(s). Thelattice graph is constructed by the trainer (152), and includes aplurality of paths that when traversed form synthetic phrases orsentences. The trainer (152) is configured to traverse the paths of thegraphs to generate the synthetic sentences. In one embodiment, each pathtraversed in its entirety forms a synthetic phrase or sentence. It isunderstood that each synthetic sentence has a similar syntacticstructure to the initial training set. Accordingly, each syntheticsentence is represented by a completed path of the lattice graph, andforms a structure grammatical related in form and substance to theinitial training set.

The trainer (152) assesses the remaining synthetic data with respect tofluency, and in one embodiment assigns a fluency score to each syntheticdata, e.g. each synthetic phrase or synthetic sentence. It is understoodthat there may be a large quantity of synthetic data, and the fluencyscore facilitates narrowing the scope of the synthetic data with respectto its relationship to the training data set. In one embodiment, thetrainer (152) applies the synthetic data to a language model to identifyand filter syntactically incorrect sentences from the set of syntheticdata generated from traversal of the lattice graph. Once the filteringor removal of the grammatically improper or low value synthetic data isremoved, a set of synthetic data related to the initial training data isformed. An assessment of the formed set of synthetic data is conductedby the trainer (152) to identify the subset of synthetic data that willbe employed with the adversarial training. As shown and described inFIG. 5, the trainer (152) calculates a log likelihood score for thesynthetic data remaining in the synthetic data set after the low valuesynthetic data has been removed. The log likelihood score is amathematical value directed at a relationship of the intent of thesynthetic data with respect to the intent of the training set, and morespecifically, represents the relative value of the intent of thesynthetic data matching the training data. The trainer (152) selects anentry in the synthetic data set with the minimum log likelihood valueand merges the corresponding utterance with the training set. Thisselection maximizes the likelihood of the worst synthetic data set, e.g.synthetic utterance. Accordingly, the adversarial training of the modelis directed at combining the worst synthetic utterance with the realtraining data.

As shown and described herein, the trainer (152) functions as a tool tobuild and maintain the model (158). It is understood that the model(158) is dynamic in that it is subject to change. The classifier (154)functions as a tool in the AI platform (150) to leverage the model (158)with respect to processing. The classifier (154) applies received inputto the model (158) to predict a classification label corresponding tothe received input. The classification label corresponds to aclassification of the intent of the request, which in one embodimentdirects the classifier (154) to an appropriate library or file withinthe knowledge base (160) to process the request. In one embodiment, theclassifier (154) applies the intent to the identified library or filewithin the knowledge base (160), and generates the response data (170).Accordingly, the classifier (154) leverages the evolved model that hasbeen subject to adversarial training, with the evolved model to identifythe intent of the input, which includes classification of a semanticmeaning of the input.

As shown and described, the trainer (152) manages evolution and trainingof the model (158). The classifier (154) leverages the model (158) tocreate response output (170), which includes classifying the intent ofreceived input, and uses the intent classification to identify anappropriate file or library within the knowledge base (160) forreceiving and processing the input. Accordingly, the trainer (152)dynamically maintains the model (158), and the classifier (154) uses thedynamically maintained model (158) to processes the received input andgenerate corresponding output.

The trainer (152) and classifier (154) function to dynamically maintainand leverage one or more classification models (158) to facilitategenerating or identifying semantically related response data, e.g.response data semantically related to the intent of received input. Asshown and described, the model (158) may be a library (166) containing aplurality of models (166A)-(166N), in which case the classifier (154)conducts a preliminary assessment of the input to identify anappropriate model (158) for processing the input. Regardless of themanner in which the model (158) is identified or selected, the modelevaluates the input, e.g. received communication, and assigns aclassification to the intent of the input, with the assignedclassification corresponding to the evaluated communication. Theclassification assignment aligns similarly related textual data. Morespecifically, the classification assigns the evaluated communication toa mathematically and proximally related library or file in the knowledgebase (160). Accordingly, the classifier (154) and the identified model(158) identify a relationship between the evaluated communication andone or more libraries or files in the knowledge base (160).

As shown and described, the original intent model (158) is augmentedwith synthetic data and subject to adversarial training. It isunderstood that there is an abundant quantity of the generated syntheticdata, and processing all of the synthetic data is a burden. One or morethresholds may be applied to narrow the set of synthetic data. Forexample, in one embodiment, a first threshold is applied with respect tosampling synthetic data, and a second threshold is applied to a secondsubset within the sampling of the applied first threshold. In oneembodiment, the second subset is a nearness qualifier, such that itidentifies synthetic data within the sampling that is mathematicallyclose or proximal to the received input. The log likelihood value isapplied to mathematically assess the proximity of the synthetic data tothe received input. The selection of the synthetic data within theminimum log likelihood value is directed at optimizing the worstsynthetic data within the sample so as to improve the relationship ofsynthetic data within the sample that is closer to the received andevaluated input. Accordingly, the weakest synthetic data in the sampleis optimized to bring new data into the model training data.

The AI platform (150), also referred to herein as an informationhandling system, employs several tools, e.g. sub-engines, to support thedescribed data processing. These tools include the trainer (152) and theclassifier (154). Types of information handling systems that can utilizesystem (110) range from small handheld devices, such as handheldcomputer/mobile telephone (180) to large mainframe systems, such asmainframe computer (182). Examples of handheld computer (180) includepersonal digital assistants (PDAs), personal entertainment devices, suchas MP4 players, portable televisions, and compact disc players. Otherexamples of information handling systems include pen, or tablet,computer (184), laptop, or notebook, computer (186), personal computersystem (188), and server (190). As shown, the various informationhandling systems can be networked together using computer network (105).Types of computer network (105) that can be used to interconnect thevarious information handling systems include Local Area Networks (LANs),Wireless Local Area Networks (WLANs), the Internet, the Public SwitchedTelephone Network (PSTN), other wireless networks, and any other networktopology that can be used to interconnect the information handlingsystems. Many of the information handling systems include nonvolatiledata stores, such as hard drives and/or nonvolatile memory. Some of theinformation handling systems may use separate nonvolatile data stores(e.g., server (190) utilizes nonvolatile data store (190 a), andmainframe computer (182) utilizes nonvolatile data store (182 a). Thenonvolatile data store (182 a) can be a component that is external tothe various information handling systems or can be internal to one ofthe information handling systems.

The AI platform (150) is local to the server (110). In some illustrativeembodiments, the server (110) may the IBM Watson™ system available fromInternational Business Machines Corporation of Armonk, N.Y., which isaugmented with the mechanisms of the illustrative embodiments describedhereafter. Although only two tools, e.g. the trainer (152) and theclassifier (154), are shown and described herein, the quantity shouldnot be considered limiting. Though shown as embodied in or integratedwith the server (110), the AI platform (150) and the associated toolsmay be implemented in a separate computing system (e.g., 190) that isconnected across network (105) to the server (110). Wherever embodied,the trainer (152) and classifier (154) function to dynamically maintainone or more intent classification models (158), assess contextualanalysis of received input with respect to classification model(s)(158), and apply the input to a library of documents (162) or files(164) in the knowledge base (160) that corresponds to the identifiedintent.

An Application Program Interface (API) is understood in the art as asoftware intermediary between two or more applications. With respect tothe NL processing system shown and described in FIG. 1, one or more APIsmay be utilized to support one or more of the tools (152)-(154) andtheir associated functionality. Referring to FIG. 2, a block diagram(200) is provided illustrating the NL processing tools and theirassociated APIs. As shown, the tools are embedded within the knowledgeengine (205), with the tools including the trainer (210) associated withAPI₀ (212) and the classifier (220) associated with API₁ (222). Each ofthe APIs may be implemented in one or more languages and interfacespecifications. API₀ (212) provides dynamic maintenance of the intentmodel(s), including generating and assessing synthetic data, andselecting a subset of the synthetic data for application to theadversarial training. API₁ (222) provides input processing with respectto an appropriately identified model and corresponding library,document, or file identification. As shown, each of the APIs (212) and(222) are operatively coupled to an API orchestrator (260), otherwiseknown as an orchestration layer, which is understood in the art tofunction as an abstraction layer to transparently thread together theseparate APIs. In one embodiment, the functionality of the separate APIsmay be joined or combined. As such, the configuration of the APIs shownherein should not be considered limiting. Accordingly, as shown herein,the functionality of the tools may be embodied or supported by theirrespective APIs.

Referring to FIG. 3, a flow chart (300) is provided to illustrate aprocess for generating synthetic utterances to expand training data. Thesynthetic utterances represent additional training data to be applied toa classifier. In the case of text based data, the classifier is a textclassifier, and in the case of image based data, the classifier is animage classifier. Although the following description is directed attextual data and associated classification models, the scope of theembodiments should not be limited to textual data, and in one embodimentmay be applied to image or graphic data. As shown and described herein,the classifier functions as a model to classify the intent of receivedata. The classifier is dynamic and is subject to modification as it isexposed to training or exposure to data.

As shown, a set of sentences are provided to train the classifier. Theset of sentences are identified and the variable X_(Total) is assignedto represent the quantity of sentences in the set (302), and anassociated sentence counting variable is initialized (304). For eachtraining sentence, sentence_(X), the intent, intent_(X), of the sentenceis identified (306) and one or more paraphrase terms for the intent areidentified (308). As described below, the training sentences arereplaced with one or more paraphrase terms to create synthetic data. Theparaphrase terms may be substitution, e.g. perturbation, of one or morewords in the training sentence. The quantity of paraphrase terms forsentence_(X) assigned to the variable Y_(Total) (310). In oneembodiment, a paraphrase database is leveraged to pair source terms inthe training sentence(s) with target term(s). Each paraphrase is a pairof source and target terms with an associated score. There are threetypes of paraphrases in the database, including: lexicon-level,phrase-level, and syntactic. The lexicon level is a paraphraserelationship of two words. The phrase-level is a paraphrase relationshipof two phrases, e.g. multiple words. The syntactic is a paraphraserelationship of two phrases with some parts as part-of-speech (POS)tags, where any belonging words can be fit into the paraphrases. Thescore is a count-based score for each source-target pair. Accordingly,for each training sentence_(X), the adversarial perturbation is aparaphrase replacement of the original training sentences on up to threelevels, including lexical, phrasal, and syntactic.

Given training sentence_(X) and identified paraphrase terms Y_(Total), alattice-based algorithm is leveraged to generate a new set ofparaphrasing, e.g. synthetic, utterances (312). The algorithm builds alattice graph with paths, and each path being a complete syntheticutterance or sentence. The lattice graph has an expanded search space.Referring to FIG. 4, a diagram (400) is provided to illustrate anexample lattice graph. In this example, the training sentence is show at(410) as “How do I find a web address for a company”. Each path sharesthe same starting point (420) and ending point (430), and each pathforms a complete sentence or utterance. In one embodiment, eachsynthetic sentence includes synonymous terms to the trainingsentence_(X). The lattice graph represents an increased search space forcandidates. In one embodiment, for each training sentence, more than 500synthetic sentences or utterances, hereinafter referred to asutterances, are generated. As the lattice graph in this example istraversed, seven example synthetic utterances are provided. Thegenerated synthetic utterances have a close semantic relationship andsimilar syntactic structures with at least the training sentence (410).Each of the paths in the lattice is traversed to build a plurality ofsynthetic utterances (314). The variable Z_(Total) is assigned to thequantity of synthetic utterances formed from traversing the paths of thelattice (316). It is understood that in one embodiment duplicationsynthetic utterances may have been developed. All duplicate utterancesin the set of generated utterances are removed (318). In the examplelattice shown in FIG. 4, the variable Z_(Total) is assigned to theinteger seven. Accordingly, synthetic utterances are formed and subjectto an accounting.

It is understood that the synthetic utterances may include low valueterms that are not necessary for the classifier training. Following theaccounting at step (318), low value terms are identified and selectivelyremoved from the synthetic utterances (320). Examples of low value termsinclude, but are not limited to, common words, stop words andtypographical errors. After the low value terms are removed at step(320), a set of synthetic utterances is generated for trainingsentence_(X) (322). Thereafter, the training sentence counting variableis incremented (324), and it is determined if each of the trainingsentences has been processed to generate the set of synthetic utterancesrelated to the intent, intent_(X), for training sentence_(X) (326). Anegative response to the determination is followed by a return to step(306), and a positive response concludes the process. Accordingly, andas shown, for each training sentence a lattice graph is generated and aset of synthetic utterances corresponding to the intent of the trainingsentence is formed.

Referring to FIG. 5, a flow chart (500) is provided to illustrate anadversarial training process to leverage the set of synthetic utterancesformed in FIG. 4. As shown, the variable X_(Total) is assigned torepresent the quantity of training sentences (502), and a trainingsentence counting variable is initialized (504). For trainingsentence_(X), the set of synthetic utterances are identified and thequantity of synthetic utterances is assigned to the variable Y_(Total)(506). In one embodiment, a subset of the synthetic utterances from thequantity identified at step (506) is utilized for the training. Thesubset is identified and selected, and assigned to the variableZ_(Total) (508). Each of the selected synthetic utterances, Z, issubmitted to a language model to filter out and remove syntacticallyincorrect sentences (510). The remaining synthetic utterances aresubmitted to a language model which returns language fluency score(512). Accordingly, each synthetic sentence_(Z) or syntheticutterance_(Z) is processed with respect to their fluency score.

The scores assessed at step (512) are employed to identify and selectdata to be employed in the adversarial training of the model. Syntheticsentence or utterances that fall below a scoring threshold are removedfrom the set (514). In one embodiment, a similarity metric, such as alanguage model or a cosine-similarity score, is applied to identify theK nearest synthetic utterances with respect to the intent of trainingsentence_(X). In one embodiment, the threshold is a configurable value.Similarly, in one embodiment, the threshold is configured with respectto quantity. Accordingly, regardless of the threshold characteristics, asubset of the synthetic utterances remains and is assigned to thevariable K_(Total) (516).

Each of the remaining synthetic utterances is subject to a loglikelihood computation with respect to the intent of the trainingsentence, sentence_(X), (518). Using the log likelihood computation, thesynthetic utterance, utterance_(K), with the minimum log likelihoodvalue is selected and identified (520). This identification reflectsoptimizing the weakest synthetic data with respect to the intent of theassociated training sentence. Accordingly, each synthetic utterance isassessed with respect to the intent of the training sentence, and thesynthetic utterance having the minimum log likelihood value isidentified and selected for use in the adversarial training of themodel.

As shown, each training sentence is subject to intent evaluation andsynthetic utterance identification based on a select computationalvalue. Following step (520), the training sentence counting variable isincremented (522), and it is determined if each of the trainingsentences has been assessed to identify synthetic data for theadversarial training (524). A negative response to the determination atstep (524) is followed by a return to step (506), and a positiveresponse to the determination at step (524) concludes the syntheticutterance evaluation and identification for adversarial training. Morespecifically, following the positive response at step (524), eachtraining sentence(s) is merged with the identified and selectedsynthetic utterance (526) determined to have the minimum log likelihoodvalue. Accordingly, each training sentence is merged with the selectedsynthetic utterance for application to a corresponding classificationmodel to support and enable adversarial training.

The process shown and described in FIG. 5 is directed at identificationof synthetic data with a minimum log likelihood value, log P (y|x),wherein x is the input, and y is the output, with respect to proximityto the respective training sentence. It is understood that theclassification model is subject to change based on use and applicationof iterations. In one embodiment, a counter is utilized to track andlimit a number of iterations for model training. For each incrementaluse, the model returned at step (526) is utilized at step (512) forcontinued training and evolution of the model. Accordingly, thesynthetic utterances are selectively identified and applied to theevolving intent classification model until such time as the counterlimit is reached or when the model is determined to be stable, e.g.changes are insignificant.

Referring to FIG. 6, a flow chart (600) is provided to illustrate anexemplary process for managing evolution of the classification modelshown and described in FIG. 5. It is understood that the adversarialtraining of the classification model is dynamic, and subject to change.The variable M₀ represents that initial intent classification model(602), and the variable N_(Total) represents the number of training sets(604). A training set counting variable is initialized (606). Realtraining data, X_(N), is identified (608), and synthetic data, X′_(N),is generated (610). The real training data, X_(N), and the syntheticdata, X′_(N), are applied to the initial intent classification model,M₀, (612). In one embodiment the synthetic data with the minimum loglikelihood value is merged with the training data, referred to herein asdata augmentation. The model, M₀, is modified to reflect application ofthe training data and the selected synthetic data (614). Application ofthe synthetic data introduces extra information, e.g. noise, to addrobustness to the intent classification model. The modified intentclassification model, also referred to herein as an updatedclassification model, reflects an incremental change in the model and isreferred to herein as Model_(N) reflecting the training set with theadversarial data, e.g. the synthetic data with the minimum loglikelihood value.

Following the model modification, the training set variable isincremented (616), and it is determined if each of the training sets andcorresponding synthetic utterance data have been applied to theclassification model (618). A negative response to the determination atstep (618) is followed by a return to steps (608) and (610) for furtherevaluation and application of data to the current version of the intentclassification model, and a positive response concludes adversarialtraining of the intent classification model. The most recent version ofthe intent classification model, Model_(N-1) is returned (620), or inone embodiment identified. Accordingly, as shown herein, the intentclassification model is subject to an incremental adversarial trainingprocess by incorporating adversarial synthetic data into the modeltraining data.

The purpose and goal of the intent classification model is to use anintent classification that has been subject to adversarial training tolabel intent from utterances. Application of synthetic data to the modelenables the model to become more robust. Referring to FIG. 7, a flowchart (700) is provided to illustrate application of the intentclassification model. As shown, input is received or detected (702). Inone embodiment, the input is a text or image. Similarly, in oneembodiment, the input is natural language (NL) that is subject toprocessing, e.g. natural language processing (NLP). The received textinput or the input converted to text is presented or otherwise receivedby the intent classification model (704), and a corresponding intent ofthe received input is identified (706). In one embodiment, theidentified intent corresponds to the topic of the received input. Usingthe intent, a classification label is applied to the received input(708), and a library or corpus corresponding to the classification labelis leveraged to return accurate response data with respect to thereceived input (710). Accordingly, trained intent classification modelis applied to the received input to generate accurate and reflectiveresponse data.

As shown and described in FIGS. 1-7, the intent classification model issubject to adversarial training and modification beyond the initialtraining, with the adversarial training including real training data andselect synthetic training data. As input is received, the model isconsulted to generate output. At the same time, augmentation of themodel may take place dynamically with the received input, and applied tothe intent classification model to continue the evolution andadversarial training of the model. For example, the received input maybe used to generate new synthetic data from which a new subset of thesynthetic data may be added to the training set for the adversarialtraining. Accordingly, the intent classification model is subject todynamic modification with respect to model training, and in oneembodiment with respect to received input.

The block diagrams and flow charts shown herein may also be in the formof a computer program device for use with an intelligent computerplatform in order to facilitate NLU and NL processing. The device hasprogram code embodied therewith. The program code is executable by aprocessing unit to support the described functionality.

As shown and described herein, the supported embodiments may be in theform of a system with an intelligent computer platform for dynamicallyintegrated content processing with classification modeling. Embodimentsmay also be in the form of a computer program device for use with anintelligent computer platform in order to assist the intelligentcomputer platform to dynamically integrated content processing andclassification modeling. The device has program code embodied therewith.The program code is executable by a processing unit to support the toolsof the AI platform (150). Content processing supported by the trainer(152) and classifier (154) may be performed in accordance to slotgrammar logic (SGL) or any other form of natural language processing.

With references to FIG. 8, a block diagram (800) is providedillustrating an example of a computer system/server (802), hereinafterreferred to as a host (802) in communication with a cloud based supportsystem, to implement the system and processes described above withrespect to FIGS. 1-7. Host (802) is operational with numerous othergeneral purpose or special purpose computing system environments orconfigurations. Examples of well-known computing systems, environments,and/or configurations that may be suitable for use with host (802)include, but are not limited to, personal computer systems, servercomputer systems, thin clients, thick clients, hand-held or laptopdevices, multiprocessor systems, microprocessor-based systems, set topboxes, programmable consumer electronics, network PCs, minicomputersystems, mainframe computer systems, and file systems (e.g., distributedstorage environments and distributed cloud computing environments) thatinclude any of the above systems, devices, and their equivalents.

Host (802) may be described in the general context of computersystem-executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.Host (802) may be practiced in distributed cloud computing environments(810) where tasks are performed by remote processing devices that arelinked through a communications network. In a distributed cloudcomputing environment, program modules may be located in both local andremote computer system storage media including memory storage devices.

As shown in FIG. 8, host (802) is shown in the form of a general-purposecomputing device. The components of host (802) may include, but are notlimited to, one or more processors or processing units (804), a systemmemory (806), and a bus (808) that couples various system componentsincluding system memory (806) to processor (804). Bus (808) representsone or more of any of several types of bus structures, including amemory bus or memory controller, a peripheral bus, an acceleratedgraphics port, and a processor or local bus using any of a variety ofbus architectures. By way of example, and not limitation, sucharchitectures include Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnects (PCI) bus. Host (802) typically includes avariety of computer system readable media. Such media may be anyavailable media that is accessible by host (802) and it includes bothvolatile and non-volatile media, removable and non-removable media.

Memory (806) can include computer system readable media in the form ofvolatile memory, such as random access memory (RAM) (830) and/or cachememory (832). By way of example only, storage system (834) can beprovided for reading from and writing to a non-removable, non-volatilemagnetic media (not shown and typically called a “hard drive”). Althoughnot shown, a magnetic disk drive for reading from and writing to aremovable, non-volatile magnetic disk (e.g., a “floppy disk”), and anoptical disk drive for reading from or writing to a removable,non-volatile optical disk such as a CD-ROM, DVD-ROM or other opticalmedia can be provided. In such instances, each can be connected to bus(808) by one or more data media interfaces.

Program/utility (840), having a set (at least one) of program modules(842), may be stored in memory (806) by way of example, and notlimitation, as well as an operating system, one or more applicationprograms, other program modules, and program data. Each of the operatingsystems, one or more application programs, other program modules, andprogram data or some combination thereof, may include an implementationof a networking environment. Program modules (842) generally carry outthe functions and/or methodologies of embodiments of the adversarialtraining and dynamic classification model evolution. For example, theset of program modules (842) may include the modules configured as theAI platform, the trainer, and the classifier, as described in FIG. 1.

Host (802) may also communicate with one or more external devices (814),such as a keyboard, a pointing device, a sensory input device, a sensoryoutput device, etc.; a display (824); one or more devices that enable auser to interact with host (802); and/or any devices (e.g., networkcard, modem, etc.) that enable host (802) to communicate with one ormore other computing devices. Such communication can occur viaInput/Output (I/O) interface(s) (822). Still yet, host (802) cancommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter (820). As depicted, network adapter(820) communicates with the other components of host (802) via bus(808). In one embodiment, a plurality of nodes of a distributed filesystem (not shown) is in communication with the host (802) via the I/Ointerface (822) or via the network adapter (820). It should beunderstood that although not shown, other hardware and/or softwarecomponents could be used in conjunction with host (802). Examples,include, but are not limited to: microcode, device drivers, redundantprocessing units, external disk drive arrays, RAID systems, tape drives,and data archival storage systems, etc.

In this document, the terms “computer program medium,” “computer usablemedium,” and “computer readable medium” are used to generally refer tomedia such as main memory (806), including RAM (830), cache (832), andstorage system (834), such as a removable storage drive and a hard diskinstalled in a hard disk drive.

Computer programs (also called computer control logic) are stored inmemory (806). Computer programs may also be received via a communicationinterface, such as network adapter (820). Such computer programs, whenrun, enable the computer system to perform the features of the presentembodiments as discussed herein. In particular, the computer programs,when run, enable the processing unit (804) to perform the features ofthe computer system. Accordingly, such computer programs representcontrollers of the computer system.

In one embodiment, host (802) is a node of a cloud computingenvironment. As is known in the art, cloud computing is a model ofservice delivery for enabling convenient, on-demand network access to ashared pool of configurable computing resources (e.g., networks, networkbandwidth, servers, processing, memory, storage, applications, virtualmachines, and services) that can be rapidly provisioned and releasedwith minimal management effort or interaction with a provider of theservice. This cloud model may include at least five characteristics, atleast three service models, and at least four deployment models. Exampleof such characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher layerof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some layer ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based email). Theconsumer does not manage or control the underlying cloud infrastructureincluding network, servers, operating systems, storage, or evenindividual application capabilities, with the possible exception oflimited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting for loadbalancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes.

Referring now to FIG. 9, an illustrative cloud computing network (900).As shown, cloud computing network (900) includes a cloud computingenvironment (950) having one or more cloud computing nodes (910) withwhich local computing devices used by cloud consumers may communicate.Examples of these local computing devices include, but are not limitedto, personal digital assistant (PDA) or cellular telephone (954A),desktop computer (954B), laptop computer (954C), and/or automobilecomputer system (954N). Individual nodes within nodes (910) may furthercommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment (900) to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices (954A-N)shown in FIG. 9 are intended to be illustrative only and that the cloudcomputing environment (950) can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 10, a set of functional abstraction layers (1000)provided by the cloud computing network of FIG. 9 is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 10 are intended to be illustrative only, and the embodiments arenot limited thereto. As depicted, the following layers and correspondingfunctions are provided: hardware and software layer (1010),virtualization layer (1020), management layer (1030), and workload layer(1040). The hardware and software layer (1010) includes hardware andsoftware components. Examples of hardware components include mainframes,in one example IBM® zSeries® systems; RISC (Reduced Instruction SetComputer) architecture based servers, in one example IBM pSeries®systems; IBM xSeries® systems; IBM BladeCenter® systems; storagedevices; networks and networking components. Examples of softwarecomponents include network application server software, in one exampleIBM WebSphere® application server software; and database software, inone example IBM DB2® database software. (IBM, zSeries, pSeries, xSeries,BladeCenter, WebSphere, and DB2 are trademarks of International BusinessMachines Corporation registered in many jurisdictions worldwide).

Virtualization layer (1020) provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers;virtual storage; virtual networks, including virtual private networks;virtual applications and operating systems; and virtual clients.

In one example, management layer (1030) may provide the followingfunctions: resource provisioning, metering and pricing, user portal,service layer management, and SLA planning and fulfillment. Resourceprovisioning provides dynamic procurement of computing resources andother resources that are utilized to perform tasks within the cloudcomputing environment. Metering and pricing provides cost tracking asresources are utilized within the cloud computing environment, andbilling or invoicing for consumption of these resources. In one example,these resources may comprise application software licenses. Securityprovides identity verification for cloud consumers and tasks, as well asprotection for data and other resources. User portal provides access tothe cloud computing environment for consumers and system administrators.Service layer management provides cloud computing resource allocationand management such that required service layers are met. Service LayerAgreement (SLA) planning and fulfillment provides pre-arrangement for,and procurement of, cloud computing resources for which a futurerequirement is anticipated in accordance with an SLA.

Workloads layer (1040) provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include, but are notlimited to: mapping and navigation; software development and lifecyclemanagement; virtual classroom education delivery; data analyticsprocessing; transaction processing; and content processing.

While particular embodiments have been shown and described, it will beobvious to those skilled in the art that, based upon the teachingsherein, changes and modifications may be made without departing from theembodiments and their broader aspects. Therefore, the appended claimsare to encompass within their scope all such changes and modificationsas are within the true spirit and scope of the embodiments. Furthermore,it is to be understood that the embodiments are solely defined by theappended claims. It will be understood by those with skill in the artthat if a specific number of an introduced claim element is intended,such intent will be explicitly recited in the claim, and in the absenceof such recitation no such limitation is present. For non-limitingexample, as an aid to understanding, the following appended claimscontain usage of the introductory phrases “at least one” and “one ormore” to introduce claim elements. However, the use of such phrasesshould not be construed to imply that the introduction of a claimelement by the indefinite articles “a” or “an” limits any particularclaim containing such introduced claim element to the embodimentscontaining only one such element, even when the same claim includes theintroductory phrases “one or more” or “at least one” and indefinitearticles such as “a” or “an”; the same holds true for the use in theclaims of definite articles.

The present embodiments may be a system, a method, and/or a computerprogram product. In addition, selected aspects of the presentembodiments may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and/or hardwareaspects that may all generally be referred to herein as a “circuit,”“module” or “system.” Furthermore, aspects of the present embodimentsmay take the form of computer program product embodied in a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent embodiments. Thus embodied, the disclosed system, a method,and/or a computer program product is operative to improve thefunctionality and operation of a machine learning model based on patterndissection of content and associated classification modeling andprocessing.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a dynamic or static random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), a magnetic storage device, a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present embodiments may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server or cluster of servers. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present embodiments.

Aspects of the present embodiments are described herein with referenceto flowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to the disclosedembodiments. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments. In this regard, each block in the flowchart or blockdiagrams may represent a module, segment, or portion of instructions,which comprises one or more executable instructions for implementing thespecified logical function(s). In some alternative implementations, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. It will also be noted that each block of the block diagramsand/or flowchart illustration, and combinations of blocks in the blockdiagrams and/or flowchart illustration, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts or carry out combinations of special purpose hardware and computerinstructions.

It will be appreciated that, although specific embodiments have beendescribed herein for purposes of illustration, various modifications maybe made without departing from the spirit and scope of the embodiments.In particular, the natural language processing may be carried out bydifferent computing platforms or across multiple devices. Furthermore,the data storage and/or corpus may be localized, remote, or spreadacross multiple systems. Accordingly, the scope of protection of theembodiments are is limited only by the following claims and theirequivalents.

What is claimed is:
 1. A computer system comprising: a processing unitin communication with memory; an artificial intelligence (AI) platformin communication with the processing unit, the AI platform to supportnatural language processing, including: a classifier, using naturallanguage understanding (NLU), to evaluate received input including:process the evaluated input against an adversarial training set;leverage the adversarial training set, and predict a classificationlabel of the received input; use the predicted classification label toidentify a corresponding classification model of the received input;present the received input to the classification model; identify, by theclassification model an intent corresponding to the received input, andidentify a corpus corresponding to the classification model; and executethe identified classification model, including using the received inputas characteristic data to the classification model; and response datagenerated from the classification model execution, wherein the responsedata reflects the received input.
 2. The system of claim 1, furthercomprising the classifier to assign the predicted classification labelto the input and classify a semantic meaning of the received input. 3.The system of claim 1, further comprising a trainer operatively coupledto the classifier, the trainer to convert the generated response data tomodel training data, and dynamically augment the identifiedclassification model with the converted model training data, includemodify the identified classification model with the received input andthe generated response data.
 4. The system of claim 3, wherein thedynamically augmented classification model functions as a NLU domainclassification index.
 5. A computer program product comprising acomputer readable storage medium having computer readable program codeembodied therewith, the program code being executable by a processor tosupport natural language processing, including program code to: usenatural language understating (NLU) to evaluate received input, andprocess the evaluated input against an adversarial training set;leverage the adversarial training set, and predict a classificationlabel of the received input; use the predicted classification label toidentify a corresponding classification model of the received input;present the input to the classification model; identify an intentcorresponding to the received to the received input, and identify acorpus corresponding to the classification model; and execute theidentified classification model, including using the received input ascharacteristic data to the classification model; response data generatedfrom the classification model execution, wherein the response datareflects the received input.
 6. The computer program product of claim 5,further comprising program code to assign the predicted classificationlabel to the input and classify a semantic meaning of the receivedinput.
 7. The computer program product of claim 5, further comprisingprogram code to convert the generated response data to model trainingdata, and dynamically augment the identified classification model withthe converted model training data, including program code to modify theidentified classification model with the received input and thegenerated response data.
 8. The method of claim 7, wherein thedynamically augmented classification model includes program code tofunction as a NLU domain classification index.
 9. A method forprocessing natural language, comprising: using natural languageunderstanding (NLU), evaluating received input, and processing theevaluated input against an adversarial training set; leveraging theadversarial training set, and predicting a classification label of thereceived input; using the predicted classification label to identify acorresponding classification model of the received input; presenting thereceived input to the classification model; identifying, by theclassification model an intent corresponding to the received input, andidentifying a corpus corresponding to the classification model;executing the identified classification model, including using thereceived input as characteristic data to the classification model; andresponse data generated from the classification model execution, whereinthe response data reflects the received input.
 10. The method of claim9, further comprising assigning the predicted classification label tothe input and classifying a semantic meaning of the received input. 11.The method of claim 9, further comprising converting the generatedresponse data to model training data, and dynamically augmenting theidentified classification model with the converted model training data,include modifying the identified classification model with the receivedinput and the generated response data.
 12. The method of claim 11,wherein the dynamically augmented classification model functions as aNLU domain classification index.